Device and computer-implemented method for data-efficient active machine learning

ABSTRACT

A device and a computer-implemented method for data-efficient active machine learning. Annotated data are provided. A model is trained for a classification of the data as a function of the annotated data. The model trained in this way is calibrated, as a function of the annotated data, with regard to confidence for a correctness of the classification of the annotated data by the model. For unannotated data, the confidence for the correctness of the classification of the unannotated data is determined, using the model calibrated in this way. The unannotated data for the active machine learning whose confidence satisfies a criterion is acquired from the unannotated data.

CROSS REFRENCE

The present application claims the benefit under 35 U.S.C. §102020200340.8 filed on Jan. 14, 2020, which is expressly incorporatedherein by reference in its entirety.

BACKGROUND INFORMATION

Monitored learning is used as the basis for many applications, includingthe classification of camera images, the detection of objects in sensormeasurements, or for voice-to-text conversions.

Monitored learning usually requires a large set of so-called trainingdata that include labels, i.e., annotations. Obtaining these trainingdata may be very complicated.

SUMMARY

In accordance with an example embodiment of the present invention, acomputer-implemented method for active machine learning provides thatannotated data are provided, a model being trained for a classificationof the data as a function of the annotated data, the model trained inthis way being calibrated, as a function of the annotated data, withregard to confidence for a correctness of the classification of theannotated data by the model, for unannotated data, the confidence forthe correctness of the classification of the unannotated data beingdetermined, using the model calibrated in this way, and the unannotateddata for the active machine learning whose confidence satisfies acriterion being acquired from the unannotated data. The overall requiredannotation effort is thus reduced.

A set of unannotated data is preferably provided, a subset beingselected from the set of unannotated data, the annotated data beingdetermined from the subset by in particular manual, semi-automatic, orautomatic annotation of unannotated data. Given an initially unlabeled,i.e., unannotated, random sample, only a small fraction of the data islabeled, i.e., annotated, in the first step. This subset may be randomlyselected, for example.

The subset preferably includes the acquired unannotated data for theactive machine learning. The next fraction to be annotated is thusselected with the aid of a suitable acquisition function.

The confidence may define the confidence as a function of at least oneprobability that the classification by the model is correct. In thisway, particularly well-suited data are selected for the training.

For a sample from the unannotated data, the criterion preferably definestwo classes having the highest probability compared to the other classesin which the sample is classifiable by the model, the samples for whicha difference between the probabilities of the two classes exceeds athreshold value being acquired from the unannotated data. Thisrepresents a quicker and better acquisition function for deep neuralnetworks in particular. For example, the trained deep neural network isinitially calibrated with the aid of temperature scaling, and aconfidence difference is subsequently used as an acquisition function.

The model is preferably iteratively trained and calibrated, a checkbeing made as to whether an abort criterion is satisfied, and the activemachine learning being ended when the abort criterion is satisfied. Thisprogressively improves the accuracy.

The abort criterion may define a reference for an accuracy of theclassification of annotated or unannotated data by the model, the abortcriterion being satisfied when the accuracy of the classificationreaches or exceeds the reference. Iterative training is thus carried outuntil the desired accuracy is achieved.

At least one parameter of the model is preferably calibrated as afunction of an expected proportion of correct classifications for theannotated data and an empirical proportion of correct classifications ofthe annotated data.

Unannotated data are preferably randomly selected in a first iterationof the method for a determination of the annotated data.

Preferably only data that are not already acquired for the subset areselected from the unannotated data for the subset.

A device for active machine learning is designed to carry out themethod, in accordance with an example embodiment of the presentinvention.

Further advantageous specific embodiments result from the followingdescription and the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic illustration of a device for active machinelearning, in accordance with an example embodiment of the presentinvention.

FIG. 2 shows steps in a method for active machine learning, inaccordance with an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Device 100 for active machine learning in accordance with an exampleembodiment of the present invention, illustrated in FIG. 1 is designedto carry out a method described below.

Device 100 includes a processing device 102, in particular one ormultiple processors, and at least one memory for data. Device 100includes a model 104.

Device 100 may be designed for digital image processing in which digitalimages are classified by model 104. In the following discussion, anacquisition function that is based on model 104 is used. In the example,model 104 is a single deep neural network. Model 104 may also be formedfrom multiple in particular deep neural networks, or may have some otherarchitecture.

A system 106 for machine learning may include device 100, a detectiondevice 108 for detecting digital images, and a control device 110 for amachine. System 106 may be designed to control the machine into a classof multiple classes as a function of a classification of a detecteddigital image. The machine may be a vehicle or a robot. Instead of aclassification of digital images, classification for a detection ofobjects in sensor measurements or for voice-to-text conversions may beprovided. Instead of detecting digital images, general sensormeasurements, such as 3D sensor measurements by radar or LIDAR sensors,for example, may be used.

A computer-implemented method for active machine learning is describedwith reference to FIG. 2. Model 104 is trained using the method, and maybe subsequently used in system 106. The method assumes that thearchitecture of model 104 is fixed and parameters of model 104 arealready initialized. In the example, model 104 is a deep neural networkwhose hyperparameter defines an input layer, an output layer, and aplurality of concealed layers in between. The parameters of the deepneural network are determined as follows.

Deep neural networks in particular may be poorly calibrated. Todetermine a suitable calibration, the expected proportion of correctpredictions is compared to the empirical proportion of correctpredictions, based on the confidence. The confidence that is output bythe deep neural network does not match the empirical accuracy on a testrandom sample.

In the example, the temperature scaling method is used for thecalibration. This is described in Guo, C., Pleiss, G., Sun, Y., &Weinberger, K. Q. (August 2017), “On calibration of modern neuralnetworks,” Proceedings of the 34th International

Conference on Machine Learning, Volume 70 (pp. 1321-1330), JMLR.org.

This means that the calibration is carried out after a training of model104. After the calibration, the confidence of the network matches theempirical accuracy better than prior to the calibration.

In the example, a difference of the two classes having the highestprobability is formed as described in Luo, Tong, et al., “Activelearning to recognize multiple types of plankton,” Journal of MachineLearning Research 6 (April 2005): 589-613.

The difference between the two classes is used in the method as anacquisition function. This acquisition function is better and quickerthan the formation of an ensemble average value, for example. Thecalibrated difference therefore yields better results.

The method is based on monitored learning. Training data includinglabels, i.e., annotations, are necessary for monitored learning. Theseannotations are used as target outputs to allow application of anoptimization algorithm. Depending on the application, the creation ofsuch a training data set is very complicated. The labeling of 3D sensormeasurements, for example of point clouds that are deliverable by aradar or LIDAR sensor, is very complicated and requires expertknowledge. In addition, in the field of medical imaging it may be verycomplicated to obtain training data from digital images.

The parameters of model 104, i.e., the deep artificial neural network inthe example, may be randomly initialized.

A set of unannotated data is provided in a step 202. The unannotateddata include patterns, in the example digital images or theirrepresentation. A step 204 is subsequently carried out.

A subset for determining the annotated data is selected from the set ofunannotated data in a step 204. The subset includes the acquiredunannotated data for the active machine learning.

In a first iteration of the method, the subset is randomly selected fromthe set of unannotated data in step 204.

Preferably only data that are not already acquired for the subset in aprevious iteration are selected from the unannotated data for the subsetin step 204.

A step 206 is subsequently carried out.

Annotated data are provided in a step 206. The annotated data may bedetermined from the subset by manual, semi-automatic, or automaticannotation of unannotated data. For this purpose, for example thedigital images are displayed and provided with a label by a human.Automated labeling methods are also usable.

A step 208 is subsequently carried out.

Model 104, the deep neural network in the example, is trained for aclassification of the data in step 208 as a function of the annotateddata. The training takes place, for example, with the aid of gradientdescent methods, for example ADAM, by monitored learning using theannotated data from the subset.

A step 210 is subsequently carried out.

Model 104 trained in this way is calibrated in step 210, as a functionof the annotated data, with regard to confidence for a correctness ofthe classification of the annotated data by the model. In the example,the confidence is defined as a function of at least one probability thatthe classification by the model is correct.

For example, at least one parameter of model 104 is calibrated as afunction of an expected proportion of correct classifications for theannotated data and an empirical proportion of correct classifications ofthe annotated data.

In the example, the temperature scaling is used, a perfect calibrationfor an input X and label Y being indicated as

P(Ŷ=Y|{circumflex over (P)}=p)=p,∀p∈[0.1]

where Ŷ is a classification and {circumflex over (P)} is the confidenceassociated with same.

The expected proportion of correct classifications is indicated by anexpected value thereof. The empirical proportion of correctclassifications is determined as a function of the annotated data andtheir classification by model 104. For the calibration, the parametersof model 104, i.e., of the deep neural network, are determined whichminimize an expected calibration error indicated below:

E(|P(Ŷ=Y|{circumflex over (P)}=p)−p|)

A step 212 is subsequently carried out.

The confidence for the correctness of the classification of theunannotated data is determined for unannotated data in step 212, usingthe model calibrated in this way.

The unannotated data for the active machine learning whose confidencesatisfies a criterion are acquired from the unannotated data.

In the example, active learning is used according to the “breaking ties”approach in order to improve the confidence of the multiclassclassification. Accordingly, an argmax_(p)P(p) is associated with aclass designation x in such a way that P(a) indicates the highestprobability and P(b) indicates the second-highest probability for aclass a or a class b, respectively. In this case, the data for which thedifference between P(a) and P(b) is smallest are eliminated from theunannotated data.

For a sample from the unannotated data, the criterion defines twoclasses, in the present case class a and class b, which have the highestprobability, P(a) and P(b) in the present case, compared to the otherclasses into which the sample is classifiable by the model. The samplesfor which a difference between the probabilities of two classes,difference P(a)−P(b) in the present case, exceeds a threshold value areacquired from the unannotated data. The threshold value may be definedas a function of the probabilities for the other of the classes. Forexample, the threshold value is defined by a difference between theprobabilities of two classes, which are compared to difference P(a)−P(b)between highest probability P(a) at that moment and second-highestprobability P(b) at that moment.

Model 104 is preferably iteratively trained and calibrated.

A check may be made in a step 214 as to whether an abort criterion issatisfied, and the active machine learning is ended when the abortcriterion is satisfied.

This means that in the example, steps 202 through 214 are carried outrepeatedly in this order or in some other order.

The abort criterion may define a reference for an accuracy of theclassification of annotated or unannotated data by the model. The abortcriterion is satisfied, for example, when the accuracy of theclassification reaches or exceeds the reference.

The calibration of the deep neural network may be checked, for example,when high-quality annotated data are present. For example, theconfidence of the deep neural network, for example 0.8, is compared tothe empirical accuracy. For a correct calibration, this would be 80%correct predictions, for example.

Different acquisition functions may be used for a given networkarchitecture. These acquisition functions may be compared by determiningwhich set of training data is necessary for achieving a certainaccuracy.

What is claimed is:
 1. A computer-implemented method for active machinelearning, the method comprising the following steps: providing annotateddata; training a model for a classification of data as a function of theannotated data; calibrating the trained model as a function of theannotated data, with regard to confidence for correctness of theclassification of the annotated data by the trained model; determining,for unannotated data, the confidence for correctness of a classificationof the unannotated data using the calibrated trained model calibrated;and acquiring, from the unannotated data, those of the unannotated datafor the active machine learning whose confidence satisfies a criterion.2. The method as recited in claim 1, further comprising: providing a setof unannotated data; selecting a subset from the set of unannotateddata; and determining the annotated data from the subset by manual, orsemi-automatic, or automatic annotation of unannotated data.
 3. Themethod as recited in claim 2, wherein the subset includes the acquiredunannotated data for the active machine learning.
 4. The method asrecited in claim 1, wherein the confidence is defined as a function ofat least one probability that the classification by the model iscorrect.
 5. The method as recited in claim 4, wherein, for a sample fromthe unannotated data, the criterion defines two classes having a highestprobability compared to the other classes in which the sample isclassifiable by the model, the samples for which a difference betweenthe probabilities of the two classes exceeds a threshold value beingacquired from the unannotated data.
 6. The method as recited in claim 1,wherein the model is iteratively trained and calibrated, a check beingmade as to whether an abort criterion is satisfied, and the activemachine learning being ended when the abort criterion is satisfied. 7.The method as recited in claim 6, wherein the abort criterion defines areference for an accuracy of the classification of annotated orunannotated data by the model, the abort criterion being satisfied whenthe accuracy of the classification reaches or exceeds the reference. 8.The method as recited in claim 1, wherein, for the calibrating, at leastone parameter of the model is calibrated as a function of an expectedproportion of correct classifications for the annotated data and anempirical proportion of correct classifications of the annotated data.9. The method as recited in claim 6, wherein the unannotated data arerandomly selected in a first iteration of the method for a determinationof the annotated data.
 10. The method as recited in claim 2, whereinonly data that are not already acquired for the subset are selected fromthe unannotated data for the subset.
 11. A device for active machinelearning, the device configured to: provide annotated data; train amodel for a classification of data as a function of the annotated data;calibrate the trained model as a function of the annotated data, withregard to confidence for correctness of the classification of theannotated data by the trained model; determine, for unannotated data,the confidence for correctness of a classification of the unannotateddata using the calibrated trained model calibrated; and acquire, fromthe unannotated data, those of the unannotated data for the activemachine learning whose confidence satisfies a criterion.
 12. Anon-transitory computer-readable storage medium on which is stored acomputer program including computer-readable instructions for activemachine learning, the instructions, when executed by a computer, causingthe computer to perform the following steps: providing annotated data;training a model for a classification of data as a function of theannotated data; calibrating the trained model as a function of theannotated data, with regard to confidence for correctness of theclassification of the annotated data by the trained model; determining,for unannotated data, the confidence for correctness of a classificationof the unannotated data using the calibrated trained model calibrated;and acquiring, from the unannotated data, those of the unannotated datafor the active machine learning whose confidence satisfies a criterion.